{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import DataFrame\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 处理丢失数据\n",
    "- 有两种丢失数据：\n",
    "    - None\n",
    "    - np.nan(NaN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 两种丢失数据的区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 为什么在数据分析中需要用到的是浮点类型的空而不是对象类型？\n",
    "    - 数据分析中会常常使用某些形式的运算来处理原始数据，如果原数数据中的空值为NAN的形式，则不会干扰或者中断运算。\n",
    "    - NAN可以参与运算的\n",
    "    - None是不可以参与运算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.nan + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "None + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 在pandas中如果遇到了None形式的空值则pandas会将其强转成NAN的形式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = DataFrame(data=np.random.randint(0,100,size=(7,5)))\n",
    "df.iloc[2,3] = None\n",
    "df.iloc[4,2] = np.nan\n",
    "df.iloc[5,4] = None\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pandas处理空值操作\n",
    "- isnull\n",
    "- notnull\n",
    "- any\n",
    "- all\n",
    "- dropna\n",
    "- fillna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 方式1：对空值进行过滤（删除空所在的行数据）\n",
    "    - 技术：isnull，notnull，any，all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#哪些行中有空值\n",
    "#any(axis=1)检测哪些行中存有空值\n",
    "df.isnull().any(axis=1) #any会作用isnull返回结果的每一行\n",
    "#true对应的行就是存有缺失数据的行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df.notnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.notnull().all(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#将布尔值作为源数据的行索引\n",
    "df.loc[df.notnull().all(axis=1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#获取空对应的行数据\n",
    "df.loc[df.isnull().any(axis=1)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#获取空对应行数据的行索引\n",
    "indexs = df.loc[df.isnull().any(axis=1)].index\n",
    "indexs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop(labels=indexs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 方式2：\n",
    "    - dropna：可以直接将缺失的行或者列进行删除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dropna(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 对缺失值进行覆盖\n",
    "    - fillna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.fillna(value=999) #使用指定值将源数据中所有的空值进行填充"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#使用空的近邻值进行填充\n",
    "#method=ffill向前填充，bfill向后填充\n",
    "df.fillna(axis=1,method='ffill')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 什么时候用dropna什么时候用fillna\n",
    "    - 尽量使用dropna，如果删除成本比较高，则使用fillna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 使用空值对应列的均值进行空值填充"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in df.columns:\n",
    "    #检测哪些列中存有空值\n",
    "    if df[col].isnull().sum() > 0:#说明df[col]中存有空值\n",
    "        mean_value = df[col].mean()\n",
    "        df[col] = df[col].fillna(value=np.around(mean_value))\n",
    "mean_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 面试题\n",
    "- 数据说明： \n",
    "    - 数据是1个冷库的温度数据，1-7对应7个温度采集设备，1分钟采集一次。\n",
    "\n",
    "- 数据处理目标：\n",
    "    - 用1-4对应的4个必须设备，通过建立冷库的温度场关系模型，预估出5-7对应的数据。\n",
    "    - 最后每个冷库中仅需放置4个设备，取代放置7个设备。\n",
    "    - f(1-4) --> y(5-7)\n",
    "\n",
    "- 数据处理过程：\n",
    "    - 1、原始数据中有丢帧现象，需要做预处理；\n",
    "    - 2、matplotlib 绘图；\n",
    "    - 3、建立逻辑回归模型。\n",
    "\n",
    "- 无标准答案，按个人理解操作即可，请把自己的操作过程以文字形式简单描述一下，谢谢配合。\n",
    "\n",
    "- 测试数据为testData.xlsx\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\tigal\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\openpyxl\\worksheet\\_reader.py:312: UserWarning: Unknown extension is not supported and will be removed\n",
      "  warn(msg)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-01-27 17:00:00</td>\n",
       "      <td>-24.8</td>\n",
       "      <td>-18.2</td>\n",
       "      <td>-20.8</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-01-27 17:01:00</td>\n",
       "      <td>-23.5</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-19.8</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-01-27 17:02:00</td>\n",
       "      <td>-23.2</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-01-27 17:03:00</td>\n",
       "      <td>-22.8</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-12.2</td>\n",
       "      <td>-9.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-01-27 17:04:00</td>\n",
       "      <td>-23.2</td>\n",
       "      <td>-18.5</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-10.2</td>\n",
       "      <td>-10.8</td>\n",
       "      <td>-8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1055</th>\n",
       "      <td>2019-01-28 10:35:00</td>\n",
       "      <td>-26.2</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-28.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1056</th>\n",
       "      <td>2019-01-28 10:36:00</td>\n",
       "      <td>-26.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1057</th>\n",
       "      <td>2019-01-28 10:37:00</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1058</th>\n",
       "      <td>2019-01-28 10:38:00</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-3.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>-5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1059</th>\n",
       "      <td>2019-01-28 10:39:00</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1060 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    time     1     2     3     4     5     6     7\n",
       "0    2019-01-27 17:00:00 -24.8 -18.2 -20.8 -18.8   NaN   NaN   NaN\n",
       "1    2019-01-27 17:01:00 -23.5 -18.8 -20.5 -19.8 -15.2 -14.5 -16.0\n",
       "2    2019-01-27 17:02:00 -23.2 -19.2   NaN   NaN -13.0   NaN -14.0\n",
       "3    2019-01-27 17:03:00 -22.8 -19.2 -20.0 -20.5   NaN -12.2  -9.8\n",
       "4    2019-01-27 17:04:00 -23.2 -18.5 -20.0 -18.8 -10.2 -10.8  -8.8\n",
       "...                  ...   ...   ...   ...   ...   ...   ...   ...\n",
       "1055 2019-01-28 10:35:00 -26.2 -27.2 -28.8 -27.5  -2.0   NaN  -5.0\n",
       "1056 2019-01-28 10:36:00 -26.8 -27.5 -29.0 -27.8  -2.2   NaN  -5.0\n",
       "1057 2019-01-28 10:37:00 -27.2 -27.8 -29.0 -28.0  -2.2   NaN  -5.0\n",
       "1058 2019-01-28 10:38:00 -27.5 -27.0 -29.0 -28.0  -3.5  -3.2  -5.8\n",
       "1059 2019-01-28 10:39:00 -27.0 -27.2 -29.0 -27.8  -5.0   NaN  -7.0\n",
       "\n",
       "[1060 rows x 8 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_excel('../data/testData.xlsx').drop(labels=['none','none1'],axis=1)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1060, 8)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(927, 8)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#删除空对应的行数据\n",
    "data.dropna(axis=0).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-01-27 17:00:00</td>\n",
       "      <td>-24.8</td>\n",
       "      <td>-18.2</td>\n",
       "      <td>-20.8</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-01-27 17:01:00</td>\n",
       "      <td>-23.5</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-19.8</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-01-27 17:02:00</td>\n",
       "      <td>-23.2</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-19.8</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-01-27 17:03:00</td>\n",
       "      <td>-22.8</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>-12.2</td>\n",
       "      <td>-9.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-01-27 17:04:00</td>\n",
       "      <td>-23.2</td>\n",
       "      <td>-18.5</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-10.2</td>\n",
       "      <td>-10.8</td>\n",
       "      <td>-8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1055</th>\n",
       "      <td>2019-01-28 10:35:00</td>\n",
       "      <td>-26.2</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-28.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1056</th>\n",
       "      <td>2019-01-28 10:36:00</td>\n",
       "      <td>-26.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1057</th>\n",
       "      <td>2019-01-28 10:37:00</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1058</th>\n",
       "      <td>2019-01-28 10:38:00</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-3.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>-5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1059</th>\n",
       "      <td>2019-01-28 10:39:00</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1060 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    time     1     2     3     4     5     6     7\n",
       "0    2019-01-27 17:00:00 -24.8 -18.2 -20.8 -18.8 -15.2 -14.5 -16.0\n",
       "1    2019-01-27 17:01:00 -23.5 -18.8 -20.5 -19.8 -15.2 -14.5 -16.0\n",
       "2    2019-01-27 17:02:00 -23.2 -19.2 -20.5 -19.8 -13.0 -14.5 -14.0\n",
       "3    2019-01-27 17:03:00 -22.8 -19.2 -20.0 -20.5 -13.0 -12.2  -9.8\n",
       "4    2019-01-27 17:04:00 -23.2 -18.5 -20.0 -18.8 -10.2 -10.8  -8.8\n",
       "...                  ...   ...   ...   ...   ...   ...   ...   ...\n",
       "1055 2019-01-28 10:35:00 -26.2 -27.2 -28.8 -27.5  -2.0  -2.2  -5.0\n",
       "1056 2019-01-28 10:36:00 -26.8 -27.5 -29.0 -27.8  -2.2  -2.2  -5.0\n",
       "1057 2019-01-28 10:37:00 -27.2 -27.8 -29.0 -28.0  -2.2  -2.2  -5.0\n",
       "1058 2019-01-28 10:38:00 -27.5 -27.0 -29.0 -28.0  -3.5  -3.2  -5.8\n",
       "1059 2019-01-28 10:39:00 -27.0 -27.2 -29.0 -27.8  -5.0  -3.2  -7.0\n",
       "\n",
       "[1060 rows x 8 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#填充\n",
    "data.fillna(method='ffill',axis=0).fillna(method='bfill',axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-01-27 17:00:00</td>\n",
       "      <td>-24.8</td>\n",
       "      <td>-18.2</td>\n",
       "      <td>-20.8</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-01-27 17:01:00</td>\n",
       "      <td>-23.5</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-19.8</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-01-27 17:02:00</td>\n",
       "      <td>-23.2</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-01-27 17:03:00</td>\n",
       "      <td>-22.8</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-12.2</td>\n",
       "      <td>-9.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-01-27 17:04:00</td>\n",
       "      <td>-23.2</td>\n",
       "      <td>-18.5</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-10.2</td>\n",
       "      <td>-10.8</td>\n",
       "      <td>-8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1055</th>\n",
       "      <td>2019-01-28 10:35:00</td>\n",
       "      <td>-26.2</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-28.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1056</th>\n",
       "      <td>2019-01-28 10:36:00</td>\n",
       "      <td>-26.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1057</th>\n",
       "      <td>2019-01-28 10:37:00</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1058</th>\n",
       "      <td>2019-01-28 10:38:00</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-3.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>-5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1059</th>\n",
       "      <td>2019-01-28 10:39:00</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1060 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    time     1     2     3     4     5     6     7\n",
       "0    2019-01-27 17:00:00 -24.8 -18.2 -20.8 -18.8   NaN   NaN   NaN\n",
       "1    2019-01-27 17:01:00 -23.5 -18.8 -20.5 -19.8 -15.2 -14.5 -16.0\n",
       "2    2019-01-27 17:02:00 -23.2 -19.2   NaN   NaN -13.0   NaN -14.0\n",
       "3    2019-01-27 17:03:00 -22.8 -19.2 -20.0 -20.5   NaN -12.2  -9.8\n",
       "4    2019-01-27 17:04:00 -23.2 -18.5 -20.0 -18.8 -10.2 -10.8  -8.8\n",
       "...                  ...   ...   ...   ...   ...   ...   ...   ...\n",
       "1055 2019-01-28 10:35:00 -26.2 -27.2 -28.8 -27.5  -2.0   NaN  -5.0\n",
       "1056 2019-01-28 10:36:00 -26.8 -27.5 -29.0 -27.8  -2.2   NaN  -5.0\n",
       "1057 2019-01-28 10:37:00 -27.2 -27.8 -29.0 -28.0  -2.2   NaN  -5.0\n",
       "1058 2019-01-28 10:38:00 -27.5 -27.0 -29.0 -28.0  -3.5  -3.2  -5.8\n",
       "1059 2019-01-28 10:39:00 -27.0 -27.2 -29.0 -27.8  -5.0   NaN  -7.0\n",
       "\n",
       "[1060 rows x 8 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.drop_duplicates(keep='first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\tigal\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\openpyxl\\worksheet\\_reader.py:312: UserWarning: Unknown extension is not supported and will be removed\n",
      "  warn(msg)\n"
     ]
    }
   ],
   "source": [
    "data = pd.read_excel('../data/testData.xlsx').drop(labels=['none','none1'],axis=1)\n",
    "data['time'] = pd.to_datetime(data['time'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-01-27 17:00:00</th>\n",
       "      <td>-24.8</td>\n",
       "      <td>-18.2</td>\n",
       "      <td>-20.8</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-27 17:01:00</th>\n",
       "      <td>-23.5</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-19.8</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-27 17:02:00</th>\n",
       "      <td>-23.2</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-19.8</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>-14.5</td>\n",
       "      <td>-14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-27 17:03:00</th>\n",
       "      <td>-22.8</td>\n",
       "      <td>-19.2</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-20.5</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>-12.2</td>\n",
       "      <td>-9.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-27 17:04:00</th>\n",
       "      <td>-23.2</td>\n",
       "      <td>-18.5</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>-18.8</td>\n",
       "      <td>-10.2</td>\n",
       "      <td>-10.8</td>\n",
       "      <td>-8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-28 10:35:00</th>\n",
       "      <td>-26.2</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-28.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-28 10:36:00</th>\n",
       "      <td>-26.8</td>\n",
       "      <td>-27.5</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-28 10:37:00</th>\n",
       "      <td>-27.2</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-2.2</td>\n",
       "      <td>-5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-28 10:38:00</th>\n",
       "      <td>-27.5</td>\n",
       "      <td>-27.0</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-28.0</td>\n",
       "      <td>-3.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>-5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-28 10:39:00</th>\n",
       "      <td>-27.0</td>\n",
       "      <td>-27.2</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>-27.8</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1060 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        1     2     3     4     5     6     7\n",
       "time                                                         \n",
       "2019-01-27 17:00:00 -24.8 -18.2 -20.8 -18.8 -15.2 -14.5 -16.0\n",
       "2019-01-27 17:01:00 -23.5 -18.8 -20.5 -19.8 -15.2 -14.5 -16.0\n",
       "2019-01-27 17:02:00 -23.2 -19.2 -20.5 -19.8 -13.0 -14.5 -14.0\n",
       "2019-01-27 17:03:00 -22.8 -19.2 -20.0 -20.5 -13.0 -12.2  -9.8\n",
       "2019-01-27 17:04:00 -23.2 -18.5 -20.0 -18.8 -10.2 -10.8  -8.8\n",
       "...                   ...   ...   ...   ...   ...   ...   ...\n",
       "2019-01-28 10:35:00 -26.2 -27.2 -28.8 -27.5  -2.0  -2.2  -5.0\n",
       "2019-01-28 10:36:00 -26.8 -27.5 -29.0 -27.8  -2.2  -2.2  -5.0\n",
       "2019-01-28 10:37:00 -27.2 -27.8 -29.0 -28.0  -2.2  -2.2  -5.0\n",
       "2019-01-28 10:38:00 -27.5 -27.0 -29.0 -28.0  -3.5  -3.2  -5.8\n",
       "2019-01-28 10:39:00 -27.0 -27.2 -29.0 -27.8  -5.0  -3.2  -7.0\n",
       "\n",
       "[1060 rows x 7 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.set_index('time',inplace=True)\n",
    "data1 = data.fillna(method='ffill',axis=0).fillna(method='bfill',axis=0)\n",
    "data1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 处理重复数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>52</td>\n",
       "      <td>15</td>\n",
       "      <td>93</td>\n",
       "      <td>6</td>\n",
       "      <td>35</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>51</td>\n",
       "      <td>83</td>\n",
       "      <td>44</td>\n",
       "      <td>67</td>\n",
       "      <td>26</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "      <td>47</td>\n",
       "      <td>60</td>\n",
       "      <td>38</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>78</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>26</td>\n",
       "      <td>13</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>90</td>\n",
       "      <td>8</td>\n",
       "      <td>51</td>\n",
       "      <td>0</td>\n",
       "      <td>48</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0   1   2   3   4   5\n",
       "0  52  15  93   6  35  32\n",
       "1   1   1   1   1   1   1\n",
       "2  51  83  44  67  26  95\n",
       "3   1   1   1   1   1   1\n",
       "4  88  91  47  60  38  20\n",
       "5   1   1   1   1   1   1\n",
       "6  78   7   4  26  13  79\n",
       "7  90   8  51   0  48  80"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = DataFrame(data=np.random.randint(0,100,size=(8,6)))\n",
    "df.iloc[1] = [1,1,1,1,1,1]\n",
    "df.iloc[3] = [1,1,1,1,1,1]\n",
    "df.iloc[5] = [1,1,1,1,1,1]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1    False\n",
       "2    False\n",
       "3     True\n",
       "4    False\n",
       "5     True\n",
       "6    False\n",
       "7    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#检测哪些行存有重复的数据\n",
    "df.duplicated(keep='first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>52</td>\n",
       "      <td>15</td>\n",
       "      <td>93</td>\n",
       "      <td>6</td>\n",
       "      <td>35</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>51</td>\n",
       "      <td>83</td>\n",
       "      <td>44</td>\n",
       "      <td>67</td>\n",
       "      <td>26</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "      <td>47</td>\n",
       "      <td>60</td>\n",
       "      <td>38</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>78</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>26</td>\n",
       "      <td>13</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>90</td>\n",
       "      <td>8</td>\n",
       "      <td>51</td>\n",
       "      <td>0</td>\n",
       "      <td>48</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0   1   2   3   4   5\n",
       "0  52  15  93   6  35  32\n",
       "1   1   1   1   1   1   1\n",
       "2  51  83  44  67  26  95\n",
       "4  88  91  47  60  38  20\n",
       "6  78   7   4  26  13  79\n",
       "7  90   8  51   0  48  80"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[~df.duplicated(keep='first')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>52</td>\n",
       "      <td>15</td>\n",
       "      <td>93</td>\n",
       "      <td>6</td>\n",
       "      <td>35</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>51</td>\n",
       "      <td>83</td>\n",
       "      <td>44</td>\n",
       "      <td>67</td>\n",
       "      <td>26</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "      <td>47</td>\n",
       "      <td>60</td>\n",
       "      <td>38</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>78</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>26</td>\n",
       "      <td>13</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>90</td>\n",
       "      <td>8</td>\n",
       "      <td>51</td>\n",
       "      <td>0</td>\n",
       "      <td>48</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0   1   2   3   4   5\n",
       "0  52  15  93   6  35  32\n",
       "1   1   1   1   1   1   1\n",
       "2  51  83  44  67  26  95\n",
       "4  88  91  47  60  38  20\n",
       "6  78   7   4  26  13  79\n",
       "7  90   8  51   0  48  80"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#异步到位删除\n",
    "df.drop_duplicates(keep='first')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 处理异常数据\n",
    "- 自定义一个1000行3列（A，B，C）取值范围为0-1的数据源，然后将C列中的值大于其两倍标准差的异常值进行清洗"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.841665</td>\n",
       "      <td>0.393672</td>\n",
       "      <td>0.164372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.201122</td>\n",
       "      <td>0.934034</td>\n",
       "      <td>0.149641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.850281</td>\n",
       "      <td>0.313631</td>\n",
       "      <td>0.141282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.740926</td>\n",
       "      <td>0.390691</td>\n",
       "      <td>0.482699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.994164</td>\n",
       "      <td>0.064755</td>\n",
       "      <td>0.316808</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C\n",
       "0  0.841665  0.393672  0.164372\n",
       "1  0.201122  0.934034  0.149641\n",
       "2  0.850281  0.313631  0.141282\n",
       "3  0.740926  0.390691  0.482699\n",
       "4  0.994164  0.064755  0.316808"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = DataFrame(data=np.random.random(size=(1000,3)),columns=['A','B','C'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5853639448425497"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#制定判定异常值的条件\n",
    "twice_std = df['C'].std() * 2\n",
    "twice_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      False\n",
       "1      False\n",
       "2      False\n",
       "3      False\n",
       "4      False\n",
       "       ...  \n",
       "995    False\n",
       "996     True\n",
       "997    False\n",
       "998     True\n",
       "999     True\n",
       "Name: C, Length: 1000, dtype: bool"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['C'] > twice_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.841665</td>\n",
       "      <td>0.393672</td>\n",
       "      <td>0.164372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.201122</td>\n",
       "      <td>0.934034</td>\n",
       "      <td>0.149641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.850281</td>\n",
       "      <td>0.313631</td>\n",
       "      <td>0.141282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.740926</td>\n",
       "      <td>0.390691</td>\n",
       "      <td>0.482699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.994164</td>\n",
       "      <td>0.064755</td>\n",
       "      <td>0.316808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>992</th>\n",
       "      <td>0.010421</td>\n",
       "      <td>0.295601</td>\n",
       "      <td>0.453018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>0.662256</td>\n",
       "      <td>0.096922</td>\n",
       "      <td>0.195074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>0.892163</td>\n",
       "      <td>0.063163</td>\n",
       "      <td>0.306347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>0.211164</td>\n",
       "      <td>0.646959</td>\n",
       "      <td>0.070807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>0.706074</td>\n",
       "      <td>0.413523</td>\n",
       "      <td>0.238120</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>567 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            A         B         C\n",
       "0    0.841665  0.393672  0.164372\n",
       "1    0.201122  0.934034  0.149641\n",
       "2    0.850281  0.313631  0.141282\n",
       "3    0.740926  0.390691  0.482699\n",
       "4    0.994164  0.064755  0.316808\n",
       "..        ...       ...       ...\n",
       "992  0.010421  0.295601  0.453018\n",
       "993  0.662256  0.096922  0.195074\n",
       "994  0.892163  0.063163  0.306347\n",
       "995  0.211164  0.646959  0.070807\n",
       "997  0.706074  0.413523  0.238120\n",
       "\n",
       "[567 rows x 3 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = df.loc[~(df['C'] > twice_std)]\n",
    "df1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "d5d375f2d234d91eb9c1d04036871c68a0c11e8c2abf8991580b257f2c707712"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
